Zero-Shot Knowledge Distillation Using Label-Free Adversarial Perturbation With Taylor Approximation

Knowledge distillation (KD) is one of the most effective neural network light-weighting techniques when training data available. However, KD seldom applicable to an environment where it difficult or impossible access data. To solve this problem, a complete zero-shot (C-ZSKD) based on adversarial learning has been recently proposed, but so-called biased sample generation problem limits performance C-ZSKD. overcome limitation, paper proposes novel C-ZSKD algorithm that utilizes label-free perturbation. The proposed perturbation derives constraint squared norm gradient style by using convolution probability distributions and 2nd order Taylor series approximation. serves increase variance distribution, which makes student model learn decision boundary teacher more accurately without labeled Through analysis distribution samples embedded space, also provides insight into characteristics are for learning-based